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Abstract 

Career and technical education research often centers around 
quantitative research designs. The Delphi Technique provides a 
structured communication process designed to produce a detailed 
examination of a topic and/or problem and discussion from the 
participating group. The contributions of individuals via this tool 
produce a group perspective not otherwise attainable. This paper 
provides the foundations of the Delphi Technique, discusses its 
strengths and weaknesses, explains the use and stages followed, 
discusses panel selection, and explains how consensus among 
participants is reached. Examples of how the Delphi Technique has 
been used in career and technical education are also included. 

Introduction 

Research literature in career and technical education is replete with studies using survey 
research as a means of seeking answers to any number of different questions. On 
occasion, however, one wishes to collect opinions of experts and, to the greatest extent 
possible, achieve consensus among them. For example, consensus among educators in a 
specific area about what technologies should be taught in today’s technology-based world 
would be important to many educators and their students. This goal presents a unique 
research situation. 

Survey research was first used in 1790 with the first U.S. Census (CASRO, 2004). Due 
to fluctuations in the economy and World War II, marketing surveys became important 
later in history as a tool to investigate the needs of consumers. Companies and 
educational institutions today are still concerned about customers’ and students’ wants 
and needs, but many times are also interested in the opinions of experts in the field and 
not simply a market analysis. 
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It would be relatively simple to ask a sample of the appropriate population their opinion 
about topic X. But then, how does one achieve consensus or agreement about topic X? 
Survey research doesn't provide a vehicle through which to achieve this goal. With the 
rate of change in today’s workplace, it is necessary to investigate future trends and needs 
in career and technical education. The Delphi technique affords the researcher such an 
opportunity. 


Foundations of the Delphi Technique 

The Delphi technique had its genesis at the Rand Corporation in the early 1950s as a 
forecasting tool for the military. Its purpose is to solicit reliable responses from a panel 
of experts regarding a particular problem or dilemma. This research tool permits 
researchers to combine the reports or testimony of a group of experts into one, useful 
statement. 

Adler and Ziglio (1996) describe the Delphi method as a communication process that is 
structured to produce a detailed examination of a topic/problem and discussion from the 
participating group, but not one that forces a quick compromise. Thus, the structuring of 
group communications is an essential component of the Delphi. The contributions of 
individuals via this tool produce a group perspective not otherwise obtainable. 

Linstone and Turoff (1975) characterize the Delphi as a method for structuring a group 
communication process. This structure permits the process to be an effective way to 
allow a group of individuals, as a whole, to deal with a complex problem. Critical to this 
communication process are four factors: “some feedback of individual contributions of 
information and knowledge; some assessment of the group judgment or view; some 
opportunity for individuals to revise views; and some degree of anonymity for the 
individual responses” (Linstone & Turoff, 1975, p. 5). This research technique allows 
educators, amongst others, to communicate and effectively develop trends, needs, or 
other factors relative to a particular area of education. In selecting the most appropriate 
research tool, however, Linstone and Turoff (1975) caution the researcher to consider the 
circumstances surrounding the “necessarily associated group communication process” (p. 
6). They suggest these guiding questions: “Who is it that should communicate about the 
problem, what alternative mechanisms are available for that communication, and what 
can we expect to obtain with these alternatives” (p. 6)? Depending on the answers to 
these questions, one may then choose the Delphi as the most effective research tool for 
the study at hand. 

Typically one of the following leads to the need for using the Delphi: 

• The problem does not lend itself to precise analytical techniques but can benefit 
from subjective judgments on a collective basis. 

• More individuals are needed than can effectively interact in a face-to-face 
exchange. 

• Time and cost make frequent group meetings infeasible. 
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• Disagreements among individuals are so severe or politically unpalatable that 
the communication process must be refereed and/or anonymity assured. 

• The heterogeneity of the participants must be preserved to assure validity of the 
results, i.e., avoidance of domination by quantity or my strength of personality 
(Linstone & Turoff, 1975). 

Linstone and Turoff (1975) have provided a mass of information about the Delphi 
technique. The Delphi exists in two basic forms: (1) The Paper-and-Pencil Version 
(Conventional Delphi), and (2) The Delphi Conference. The Conventional Delphi 
includes questions being sent to a group and, based on their responses, a new 
questionnaire is developed and disseminated to the same group. This continues through 
several rounds, providing the group the opportunity to reevaluate their own answers 
based upon examination of the group response. The Delphi Conference uses a computer 
program to compile and distribute the group results. The main advantage of this method 
is that there is no delay in summarizing feedback from each round and returning the 
group responses. Essential components of the Delphi technique, to be discussed in more 
detail later, include the communication process, a group of experts, and essential 
feedback. 

There are four individual phases in the Delphi process. The first phase explores the 
subject being researched, giving participants the opportunity to contribute information 
they feel is appropriate. The second phase moves to determine an understanding of how 
the entire group views the issue. If significant disagreement is determined, the third 
phase is used to explore that disagreement and determine reasons for differences. The 
fourth phase is a final evaluation of all gathered information (Linstone & Turoff, 1975). 

There are different structures within the Delphi method. Three include the Policy Delphi 
Model, the Trend Model (Turoff, 1970), and the Structural Model (Lendaris, 1980; 
Geoffrion, 1987). The Policy Delphi works toward discovering the strongest pro and con 
arguments about differing resolutions for a specific policy issue. It does not produce a 
consensus. The Trend Model first deals with a specific trend that is of concern to the 
group, and participants project where they believe the trend will go in the next five years. 
Assumptions and uncertainties are listed when making the projections. The assumptions 
are voted on for validity. The participants are later asked to review the valid assumptions 
and re-estimate their earlier trend projections. Structural Modeling allows participants 
individually to express independent relationships/judgments, but they are all used to 
produce a group or whole model or system. This is supported by Helmer (1977) who 
notes that the Delphi is a useful communication method among an expert panel that in 
turn facilitates the formation of a group judgment. 

Cornish (1977) noted that the Delphi has been used widely as a way to forecast trends in 
technology, education, and other fields. Some studies specific to career and technical 
education include Wilhelm (1999) who used the Delphi technique to determine entry- 
level workplace skills and competencies. LaBonty and Scott (1993) used the Delphi 
method to investigate perceptions of business teacher educators. Bmening, et al. (2002) 
investigated attributes and characteristics of exemplary career and technical teacher 
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preparation programs. Marketing education has also used the Delphi method to decipher 
effective instructional strategies in Tennessee marketing education (McCurry, 1996). 
Farmer (1998) used the Delphi technique to determine research priorities in career and 
technical education. Wicklein (1993) identified critical issues and problems in technology 
education using a modified Delphi technique. 

Chandler (1996) used the Delphi as a foundation for making recommendations to the 
Associated Landscape Contractors of America regarding their career days program. 
Pullen (1996) used the Delphi to develop a profile of the duties of the secondary career 
and technical education supervisor predicted to be needed in the 21 st century. Flanders 
(1988) also used the Delphi to determine curriculum content for nursery/landscape course 
work in vocational agriculture courses for the 21 st century. 

Strengths and Weaknesses 

Stewart (2001) stated this regarding the utility of the Delphi, “Its capacity to capture 
those areas of collective knowledge that are held within professions but not often 
verbalized, makes it enormously useful in the field of professional education” (p. 922). 
The Delphi is exceptionally useful where the judgments of individuals are needed to 
“address a lack of agreement or incomplete state of knowledge . . . [thus] the Delphi is 
particularly valued for its ability to structure and organize group communication” 
(Powell, 2003, p. 376). Powell reported further that the systematic control inherent in the 
Delphi design increases “an air of objectivity to the outcome” (p. 377). 

When the proper steps are followed in a Delphi study, including anonymity, the panel 
members can express their opinions freely. When the Delphi Conference is used, the 
participants can “meet” any time to provide their judgments/suggestions/op inions and 
feel as though they are a part of a group solving or discussing a particular problem. 
Turoff and Hiltz (1996) identify other advantages of the Delphi. These advantages 
include the capability of allowing individuals with the appropriate knowledge in the 
content area to have "differing perspectives and differing cognitive abilities” but still 
contribute to those parts of a complex problem that deal with their area. This provides a 
flexible way for individuals to approach difficult questions and solve complex problems. 
Turoff and Hiltz (1996) continue by stating that the most important criterion in the design 
of a Delphi study is enabling each individual "to choose the sequence in which to 
examine and contribute to the problem solving process" (p. 3). 

Every type of research has its strengths and weaknesses. It is obvious that the 
inappropriate choice of the Delphi technique as a research tool will typically result in 
failure. Linstone and Turoff (1975) suggest the following as reasons for failure: 

• Imposing monitor views and preconceptions of a problem upon the respondent 
group by overspecifying the structure of the Delphi and not allowing for the 
contribution of other perspectives related to the problem. 

• Assuming that the Delphi can be a surrogate for all other human 
communications in a given situation. 
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• Using poor techniques of summarizing and presenting the group response and 
ensuring common interpretations of the evaluation scales utilized in the exercise. 

• Ignoring and not exploring disagreements, so that discouraged dissenters drop 
out and an artificial consensus is generated. 

• Underestimating the demanding nature of a Delphi and the fact that the 
respondents should be recognized as consultants and properly compensated for 
their time if the Delphi is not a part of their job function. 

Another weakness may include the researcher not being able to conceptualize different 
ways to examine the problem being investigated. As researchers become more creative 
in perceiving how different individuals may view the same problem in many different 
ways, this becomes less of a weakness. Also, if a researcher hastily tries to complete a 
Delphi study, thorough time for thought may not be given to the problem and consensus 
may not be obtained. 

Researchers must be careful to follow the steps in any Delphi study to make sure that the 
experts in the group have a chance to provide appropriate feedback. Time is also 
necessary so that the researcher appropriately restructures the questionnaire from round 
to round so that adequate and correct feedback is provided through the questionnaire. All 
of these weaknesses and limitations result in a poorly designed and executed Delphi 
study which will result in inadequate or incorrect data. 

Linstone and Turoff (1975) further elaborate on the potential problems of Delphi studies 
with a checklist of eight pitfalls including: discounting the future; the prediction urge, the 
simplification urge, illusory expertise, sloppy execution, optimism-pessimism bias, 
overselling, and deception. To avoid these pitfalls, Delphi researchers must make sure 
that they are open-minded, not biased, and willing to put the time into the study that it 
requires. 


Use and Stages of the Delphi Technique 

This section provides specifics as to how to use the Delphi technique to its best 
advantage. Successful use of the Delphi involves a communication procedure through 
which information/feedback/opinion is obtained from the participants. The following 
summarize stages proposed by Beech (1999): 

1. Selection of panel (respondents) and allocation of identification numbers. 

2. Construction and distribution of first questionnaire (Round One). Completion 
and return of Round One questionnaire. 

3. Collation and categorization of suggestions and construction of second 
questionnaire (Round Two). 


©2004 - Journal of Career and Technical Education, Vol. 20, No. 2, Spring, 2004 - Page 59 



4. Distribution of second questionnaire (Round Two). Completion and return of 
Round Two questionnaire. 

5. Collation of individual and group scores for each suggestion. 

6. Construction of third questionnaire (Round Three) which is similar to Round 
Two questionnaire but with individual and group scores for each suggestion 
from Round Two incorporated. 

7. Distribution of third questionnaire (Round Three). Completion and return of 
Round Three questionnaire. 

8. Recollation of individual and group scores for each suggestion. 

9. Possible further rounds of voting and possible request for rationale and 
comments for more extreme scores. 

10. Achievement of group consensus with calculation of summary statistics: 
maximum, minimum, and range of scores for each suggestion. 

11. Distribution and use of findings. 

Steps may indeed vary from these suggestions; however, the procedure remains 
consistent. For example, step two above indicates development of a questionnaire. If the 
nature of the research project is to provide an answer to a question, then indeed a 
questionnaire would be developed (i.e., What are the primary concerns regarding 
classroom management of career and technical education student teachers?). However, 
Wilhelm (2001), stated “In some Delphis the first round is structured such that the topics 
are presented to the respondents in order for them to evaluate, elaborate, or otherwise 
comment on the topics with their individual concerns, insights, criticisms, or agreement” 
(p. 14). For example, round one might ask participants: Rank the following topics in 
order of importance in preparing future career and technical education teachers: 

• Career/technical student organizations 

• Philosophy of career and technical education 

• Specific methods of instruction in program areas under career/technical 
education 

• Other, please specify. 

Panel Selection 

Careful selection of the panel of experts is the keystone to a successful Delphi study. 
Key aspects include panel selection (including experts’ qualifications) and size and 
participant commitment. Jairath and Weinstein (1994) suggest that study participants be 
experts who are knowledgeable about current information and perceptions regarding the 
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topic under investigation but are open-minded to the findings. Gibbs, Graves, and Bernas 
(2001) in their study of evaluation guidelines for multimedia software used these criteria 
to determine who should be selected and invited to participate: 

• Participants had published articles in the last five years on computer-based 
courseware design, development, or evaluation; 

• Participants had taught courses about these topics; or 

• Participants’ primary employment responsibilities related to these areas (p. 4). 

Wicklein (1993) achieved a 100% return rate in a Delphi study used to collect 
information about issues and problems in technology education. This Delphi study 
consisted of four rounds and 25 leading experts in technology education who were 
carefully selected "because the success of the Delphi relies on informed opinion," not 
random selection (p. 1050). Recognition of being selected for a particular study in which 
participants are noted as experts in their field may be encouragement enough to get 
participants to agree to provide their opinions to complete all rounds of the study. 

Thus, one might look to professional associations under the career and technical 
education umbrella such as members of the International Technology Education 
Association, the National Business Education Association, or the Association for Career 
and Technical Education. Powell (2003) also reported that heterogeneous rather than 
homogeneous groups “produce a higher proportion of high quality, highly acceptable 
solutions” (p. 379). To avoid homogeneity in career and technical education research, 
one might solicit participation from secondary educators, technical college educators, and 
college/university faculty. 

To motivate participants to remain active and complete all the rounds in a Delphi study, it 
is important they understand the goal of the study and feel they are a part of the group. 
The participants must also believe they are able to contribute valuable judgments and 
help examine the problem via discussion with their peers. If the Delphi researcher can 
locate the expert group of peers and invite them individually to participate, the likelihood 
of increased participation is stronger. Turoff and Hiltz (1996) note that “blank 
invitations” do not provide sufficient specifics about who will participate; therefore, the 
potential participants do not know if it will be a group of their peers and participation is 
usually very low. 

Anonymity is an important Delphi technique characteristic. Without the identification of 
those contributing information or making specific judgments, candor may be increased. 
Turoff and Hiltz (1996) cite these advantages to anonymity: 

• If a participant provides an idea/suggestion/judgment and in the end it is deemed 
unsuitable, no one loses face. 

• When a participant commits his/her name to an idea/suggestion/judgment, it is 
harder to reject/change it later. 
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• The idea/suggestion/judgment may be biased by who introduced it if anonymity 
is not used. 

Keeping in mind that group communication is an essential component of the Delphi, the 
anonymity will remove common biases and social interaction that normally occur in face- 
to-face group communication. 

A variety of perspectives exist regarding the number of participants needed. Many 
support the notion that the more the participants, the better (Murphy-Black, et al., 1998). 
However, Powell (2003) also stated that, “There is very little actual empirical evidence 
on the effect of the number of participants on the reliability or validity of consensus 
processes” (p. 378). Taylor-Powell (2002) stated that the number of participants in a 
Delphi study depends not only on the purpose of the study, but the diversity of the 
targeted population. Ten to 15 participants may be an adequate number for a Delphi 
study that is focused and where the participants do not vary a great deal. 

Rounds and Determining Consensus 

The Delphi “uses rounds of written questionnaires and guaranteed anonymity with 
summarized information and controlled feedback to produce a group consensus on an 
issue” (Beech, 1999, p. 283). Unlike survey research, the rounds used with the Delphi 
provide opportunity for initial feedback, collation of feedback, and distribution of 
collated feedback to participations for further review. This process requiring group 
communication is central to the strength of the Delphi. The Delphi method involves a set 
of procedures for eliciting and refining the opinions of a group (Brown, 1968; Dalkey, 
1967). In rounds two and three participants are typically asked to provide a rating based 
on a Likert-type response (i.e., 1 is never, 5 is always, etc.). Median scores are then 
developed as are means and standard deviations. In rounds two or three, “it is also 
possible to request either a rationale or clarification from experts who score items outside 
a particular range, for example +/- two points of the group median score” (Beech, 1999, 
p. 284). 

For example, if one were attempting to determine the responsibilities of a career and 
technical education supervisor using the Delphi technique, round one might start with 
questions such as, 

• “What duties related to induction of new teachers, staff development activities, 
and teacher evaluation should be performed?” or 

• “What duties relative to maintaining facilities and technology should be 
performed?” or 

• “What duties relative to budget development and records maintenance should be 
performed?” 

Participants would then be free to record as many responses to each question they 
deemed necessary. In round two, the responses to each question would be collated 
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and/or categorized if needed. The question and responses would be developed into a 
second questionnaire that provided a Likert-type response for every response 
provided. For example, a round two question might look like this: 


Duties related to the induction of new teachers, staff development activities, and 
teacher evaluation: 

1. Establish a mentoring program for first-year teachers 

and teachers new to the system. 1 2 3 4 5 

2. Create staff develop activities around program area 

needs. 1 2 3 4 5 

3. Peruse published literature on evaluation models to 

develop one appropriate for the system. 1 2 3 4 5 


In round two, then, participants would be asked to rate each item using the Likert scale 
and key to scale numbers provided. The results, vis a vis each participant’s score and the 
median for each item from round two would be computed. Participants would then be 
asked (in Round 3) to either retain their initial score or revise it up or down. An example 
follows. 


Your Median Round 3 

No. Item Score Score Score 

1. Establish a mentoring program for 
first-year teachers and teachers new to the 

system. 4 4 1 2 3 4 5 

2. Create staff develop activities around 

program-area needs. 3 4 1 2 3 4 5 

3. Peruse published literature on evaluation 
models to develop one appropriate for the 

system. 2 3 1 2 3 4 5 


Of course participants’ scores would not be revealed to the entire group, only to 
participant who owned the score. From this, round three feedbacks, one then is able to 
determine consensus. 

Knowing that participants are in agreement on the questions is another key element of the 
Delphi. Consensus or convergence of opinion is an underlying principle of the Delphi 
(McCallister, 1992). Weatherman and Swenson (1974) stated that there should be a 
convergence of opinion as individual estimations move closer to the statistical survey of 
the group responses. One may decide, then, that when 60% or more of respondents give 
the same response on an item, consensus on that item was achieved. Also, if any item 
had more than 15% change in the mean score from one round to the next, the item was 
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considered unstable and a succeeding round was needed. The median can be used to 
determine central tendency (Chandler, 1996). 

Crews and Ray (1997) utilized a modified Delphi where a list of course topics for a 
telecommunications course were provided in round one and the participants could add, 
delete, or edit from that initial list. The participants were then provided with the new list 
in round two; and they could again add, delete, or edit from that initial list. Round three 
used the Likert scale to rate the items, and round four gave the participants an opportunity 
to see the group mean and change or keep their ratings on the topics. Consensus was 
achieved only if 100% of the participants rated an item at the highest rating in the Likert 
scale provided. However, if two-thirds of the participants rated the item at the highest 
rating, common consent was achieved. If the Delphi method is looked at as outcome- 
based, the outcome is an opinion. This opinion is only as valid as the opinions of those 
experts who serve as the group providing their judgments/suggestions/op inions (Martino, 
1993). 

There has to be a clear acknowledgement about when the rounds should cease. Linstone 
and Turoff (1975) hold the view that “when something has attained a point at which is it 
explicitly definable, then progress has stopped” (p. 5). 

Summary 

The Delphi method provides different opportunities to researchers than survey research. 
Essential components of the Delphi technique include the communication process, a 
group of experts, and essential feedback. The Delphi method has been used in a variety of 
ways in government, business, and education. This manuscript has presented the key 
principles of the Delphi technique and has provided a number of examples where it has 
been used in career and technical education. “The technique benefits from being a 
democratic and structured approach that harnesses the collective wisdom of participants” 
(Powell, 2003, p. 381). Its most significant strength lies in the ability to gamer opinion 
and seek consensus among a diverse group of participants. 
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